Skip to content

benchsdk: clean up SDK surface, remove admin API key, validate CLI flags, and sanitize scaffold names - #362

Merged
dtice25 merged 5 commits into
masterfrom
devin/benchsdk-review-fixes
Aug 24, 2026
Merged

dtice25 merged 5 commits into
masterfrom
devin/benchsdk-review-fixes

Conversation

@dtice25

@dtice25 dtice25 commented Aug 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Tightens the benchsdk packages so the SDK surface is single-sourced, auth is predictable, CLI/scaffold behavior is safer, and CI only requests the benchmarks platform API key.

Changes

  • Single-source @benchsdk/client from @benchsdk/api + @benchsdk/worker.

    • Removed the duplicated packages/benchsdk/src/types.ts and the thin re-export files (metrics.ts, participants.ts, reporter.ts).
    • index.ts now re-exports the public API/worker surface, and client.ts extends BenchmarkClient from @benchsdk/api with runWorker.
  • Eliminate unsafe casts in client.ts.

    • runWorker is now typed as:
      export type BenchmarkClient = BenchmarkApiClient & {
        runWorker(options: RunWorkerOptions): Promise<RunWorkerResult>;
      };
    • createBenchmarkClient calls runWorker(apiClient, options) directly without as any.
  • Stop library code from exiting the process.

    • selectParticipants in @benchsdk/worker now throws a descriptive Error instead of calling process.exit(1).
    • Fixed the malformed JSDoc block.
  • Remove COMPUTESDK_ADMIN_API_KEY from all auth paths.

    • @benchsdk/api now checks BENCHMARKS_PLATFORM_API_KEY → COMPUTESDK_API_KEY.
    • @benchsdk/runner resolvePlatform now checks BENCHMARKS_PLATFORM_API_KEY → COMPUTESDK_API_KEY.
    • Scale scripts and coordinator use process.env.BENCHMARKS_PLATFORM_API_KEY ?? process.env.COMPUTESDK_API_KEY.
    • Removed COMPUTESDK_ADMIN_API_KEY from all benchmark workflow vault-secret allowlists.
  • Reject unknown CLI flags in @benchsdk/runner.

    • parseCliArgs collects unrecognized flags and throws Unknown flag(s): ... unless the flag is declared in BenchmarkConfig.customCliFlags.
    • Storage and AI-gateway benchmark configs declare their pass-through flags (--file-size, --dataset, --ai-gateway-iterations-cold, --ai-gateway-iterations-warm).
  • Sanitize create-bench project names.

    • Project names are lowercased, whitespace collapsed to -, invalid characters stripped, and runs of - collapsed so package.json name is always valid.
  • Fix empty-package test scripts.

    • @benchsdk/api and @benchsdk/worker now use vitest run --passWithNoTests, so pnpm test no longer fails when no test files exist.
  • Fix scale image bundle.

    • bundle:scale now emits ESM (dist/scale.mjs) and injects a createRequire shim so bundled CJS dependencies such as dotenv can still require Node built-ins at runtime.
    • Updated benchmarks/scale/Dockerfile and benchmarks/scale/scripts/start.ts to run /app/coordinator.mjs.
  • Resolve TypeScript/@types/node drift.

    • Aligned root devDependencies with the workspace packages (typescript ^5.0.0, @types/node ^20.0.0).

Verification

  • pnpm typecheck passed
  • pnpm -r --filter "./packages/*" build passed
  • pnpm -r --filter "./packages/*" test passed
  • pnpm run bundle:scale passed

Link to Devin session: https://app.devin.ai/sessions/6f1c1763ac8b47a49203e9749b5b88cd
Requested by: @dtice25


Open in Devin Review

…caffolding

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 24, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

dtice25 and others added 3 commits August 24, 2026 16:23
…flag rejection

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration devin-ai-integration Bot changed the title benchsdk: clean up SDK surface, align auth fallbacks, validate CLI flags, and sanitize scaffold names benchsdk: clean up SDK surface, remove admin API key, validate CLI flags, and sanitize scaffold names Aug 24, 2026
…ndencies

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25
dtice25 merged commit 148b420 into master Aug 24, 2026
8 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Browser Benchmark Results

# Provider Score Create Connect Navigate Release Total Status
1 Kernel 97.4 0.04s 0.06s 0.12s 0.05s 0.27s 10/10
2 Browseruse 95.8 0.18s 0.11s 0.07s 0.04s 0.42s 10/10
3 Browserbase 93.6 0.15s 0.30s 0.13s 0.13s 0.78s 10/10
4 Hyperbrowser 92.7 0.28s 0.15s 0.14s 0.10s 0.67s 10/10
5 Notte 86.6 0.19s 0.82s 0.28s 0.09s 1.50s 10/10
6 Steel 79.7 0.41s 0.80s 0.32s 0.20s 1.83s 10/10
7 Tilion 0.0 0.00s 0.00s 0.00s 0.00s 0.00s 0/10

View full run · SVG available as build artifact

@github-actions

Copy link
Copy Markdown
Contributor

Storage Benchmark Results

1MB Files

# Provider Score Download Throughput Upload Status
1 Azure Blob Storage 96.2 0.03s 293.6 Mbps 0.08s 10/10
2 Archil 96.0 0.03s 321.9 Mbps 0.38s 10/10
3 Vercel Blob 95.2 0.06s 143.4 Mbps 0.28s 10/10
4 AWS S3 95.0 0.09s 97.8 Mbps 0.20s 10/10
5 Tensorlake 94.8 0.05s 170.1 Mbps 0.59s 10/10
6 Neon 94.7 0.13s 62.9 Mbps 0.26s 10/10
7 Mosaic 94.5 0.11s 76.8 Mbps 0.46s 10/10
8 Tigris 94.5 0.10s 85.0 Mbps 0.27s 10/10
9 Cloudflare R2 92.6 0.13s 65.3 Mbps 0.46s 10/10
10 Google Cloud Storage 0.0 0.00s 0.0 Mbps 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

Copy link
Copy Markdown
Contributor

Snapshot/Fork Benchmark Results

small dataset

# Provider Score Snapshot create Fork (snapshot) Fork (live) First read Status
1 Tensorlake 99.8 0.04s 0.18s 0.23s 0.11s 1/1
2 Tigris 99.7 0.13s 0.26s 0.24s 0.20s 1/1
3 Azure-blob 98.9 0.66s 0.53s 1.48s 0.02s 1/1
4 Archil 0.0 0.00s 0.00s 0.00s 0.00s 0/1
5 AWS S3 0.0 0.00s 0.00s 0.00s 0.00s 0/1
6 Cloudflare R2 0.0 0.00s 0.00s 0.00s 0.00s 0/1
7 Mosaic 0.0 0.00s 0.00s 0.00s 0.00s 0/1
8 Neon 0.0 0.00s 0.00s 0.00s 0.00s 0/1

View full run

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Burst

# Provider Score Median TTI P95 P99 Status
1 declaw 98.6 0.14s 0.15s 0.15s 10/10
2 createos 98.6 0.14s 0.15s 0.15s 10/10
3 isorun 98.0 0.20s 0.20s 0.20s 10/10
4 daytona 97.4 0.22s 0.31s 0.31s 10/10
5 archil 97.1 0.21s 0.41s 0.41s 10/10
6 arker 96.2 0.38s 0.38s 0.38s 10/10
7 blaxel 94.9 0.46s 0.59s 0.59s 10/10
8 superserve 93.5 0.57s 0.76s 0.76s 10/10
9 vercel 93.5 0.54s 0.82s 0.82s 10/10
10 upstash 92.0 0.78s 0.84s 0.84s 10/10
11 modal 91.9 0.78s 0.87s 0.87s 10/10
12 runloop 88.8 0.69s 1.76s 1.76s 10/10
13 e2b 87.3 1.24s 1.33s 1.33s 10/10
14 beam 86.2 1.36s 1.39s 1.39s 10/10
15 tensorlake 83.0 1.40s 2.14s 2.14s 10/10
16 sandbox0 81.6 1.72s 2.01s 2.01s 10/10
17 mosaic 69.6 0.56s 6.77s 6.77s 10/10
18 cloud-run 69.5 2.49s 3.90s 3.90s 10/10
19 sail 68.9 2.17s 4.53s 4.53s 10/10
20 cloudflare 65.7 2.77s 4.42s 4.42s 10/10
21 codesandbox 63.1 3.31s 4.27s 4.27s 10/10
22 hopx 0.0 0.00s 0.00s 0.00s 0/10
23 lightning 0.0 0.00s 0.00s 0.00s 0/10
24 microsandbox 0.0 0.00s 0.00s 0.00s 0/10
25 miosa 0.0 0.00s 0.00s 0.00s 0/10
26 northflank 0.0 0.00s 0.00s 0.00s 0/10
27 opencomputer 0.0 14.16s 15.78s 15.78s 10/10
28 run-cloud 0.0 14.06s 18.87s 18.87s 10/10
29 tenki 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

Copy link
Copy Markdown
Contributor

Browser Throughput Benchmark Results

# Provider Score APS (med) Task (med) Task (p95) Screenshot Status
1 Kernel 81.7 6.14/s 1.63s 2.02s 233ms 3/3
2 Browserbase 70.3 3.66/s 2.73s 2.81s 344ms 3/3
3 Browseruse 69.0 3.42/s 2.92s 3.18s 240ms 3/3
4 Hyperbrowser 61.7 2.80/s 3.57s 9.34s 532ms 3/3
5 Steel 57.2 1.79/s 5.58s 7.45s 705ms 3/3
6 Notte 47.9 1.00/s 10.02s 10.50s 1429ms 3/3
7 Tilion 0.0 0.00/s 0.00s 0.00s 0ms 0/3

View full run · SVG available as build artifact

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Dax Benchmark Results

Provider Phases Total Prepare Clone Install Typecheck Status
archil 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
arker 7/7 74.25s 7.08s 19.62s 10.51s 32.13s 1/1 OK
beam 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
blaxel 7/7 57.41s 1.34s 13.79s 10.22s 24.20s 1/1 OK
cloud-run 2/7 0.91s 0.19s 0.00s 0.00s 0.00s 0/1 OK
cloudflare 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
codesandbox 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
createos 7/7 50.04s 2.42s 3.55s 13.88s 25.17s 1/1 OK
daytona 7/7 69.65s 5.32s 5.07s 12.68s 35.79s 1/1 OK
declaw 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
e2b 7/7 84.54s 10.04s 3.97s 20.64s 40.54s 1/1 OK
hopx -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
isorun 7/7 63.04s 4.98s 13.15s 11.28s 28.53s 1/1 OK
lightning -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
microsandbox -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
miosa -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
modal 6/7 29.02s 8.32s 2.57s 12.77s 0.00s 0/1 OK
mosaic 6/7 14.84s 1.28s 8.68s 2.59s 0.00s 0/1 OK
namespace -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
northflank -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
opencomputer 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
run-cloud -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
runloop 7/7 134.78s 32.47s 4.43s 32.62s 53.97s 1/1 OK
sail 7/7 55.36s 1.87s 2.79s 11.70s 32.98s 1/1 OK
sandbox0 7/7 176.06s 6.32s 7.77s 73.61s 63.03s 1/1 OK
superserve 7/7 113.79s 11.76s 5.80s 21.57s 69.94s 1/1 OK
tenki -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
tensorlake 7/7 96.75s 18.77s 6.15s 16.40s 47.48s 1/1 OK
upstash 7/7 59.33s 3.66s 10.69s 9.72s 25.09s 1/1 OK
vercel 7/7 69.17s 10.61s 1.99s 12.33s 37.17s 1/1 OK

View full run

@github-actions

Copy link
Copy Markdown
Contributor

AI Gateway Benchmark Results — OpenAI

# Gateway Score Cold E2E Warm TTFT Tok/sec Status
1 Pydantic AI Gateway 90.7 494ms 408ms 69.4 20/20
2 OpenAI (direct) 90.6 516ms 409ms 65.7 20/20
3 OpenRouter 89.5 516ms 396ms 60.4 20/20
4 Vercel AI Gateway 89.1 593ms 523ms 53.8 20/20
5 Concentrate AI 88.9 756ms 655ms 49.1 20/20
6 Neon AI Gateway 88.5 797ms 586ms 51.6 20/20
7 Cloudflare AI Gateway 88.1 618ms 426ms 61.1 20/20
8 ngrok AI Gateway 87.7 678ms 403ms 61.6 20/20
9 LLM Gateway 87.2 878ms 945ms 60.4 20/20

View full run · SVG available as a build artifact

@github-actions

Copy link
Copy Markdown
Contributor

AI Gateway Benchmark Results — Anthropic

# Gateway Score Cold E2E Warm TTFT Tok/sec Status
1 Cloudflare AI Gateway 89.6 627ms 517ms 52.2 20/20
2 ngrok AI Gateway 89.6 629ms 519ms 52.6 20/20
3 OpenRouter 89.4 536ms 559ms 52.3 20/20
4 Pydantic AI Gateway 89.4 591ms 565ms 51.6 20/20
5 LLM Gateway 89.2 645ms 647ms 50.3 20/20
6 Anthropic (direct) 89.1 537ms 584ms 53.3 20/20
7 Vercel AI Gateway 89.0 714ms 677ms 49.2 20/20
8 Concentrate AI 88.0 927ms 758ms 44.4 20/20
9 Neon AI Gateway 87.5 930ms 842ms 48.3 20/20
10 blazerail 0.0 0ms 0ms 0.0 0/20

View full run · SVG available as a build artifact

@github-actions

Copy link
Copy Markdown
Contributor

AI Gateway Benchmark Results — Gemini

# Gateway Score Cold E2E Warm TTFT Tok/sec Status
1 OpenRouter 86.2 1822ms 1901ms 105.2 20/20
2 Vercel AI Gateway 86.0 1999ms 1766ms 102.5 20/20
3 LLM Gateway 84.9 1941ms 2097ms 95.4 20/20
4 Neon AI Gateway 82.2 1722ms 1624ms 3.8 20/20
5 Gemini (direct) 81.0 1973ms 1836ms 3.7 20/20
6 Cloudflare AI Gateway 80.8 2005ms 1757ms 3.3 20/20
7 Concentrate AI 78.8 2359ms 2148ms 82.5 19/20
8 ngrok AI Gateway 61.3 2867ms 4174ms 1.9 19/20
9 blazerail 0.0 0ms 0ms 0.0 0/20

View full run · SVG available as a build artifact

@github-actions

Copy link
Copy Markdown
Contributor

AI Gateway Benchmark Results — Kimi

# Gateway Score Cold E2E Warm TTFT Tok/sec Status
1 Neon AI Gateway 86.0 990ms 705ms 92.9 19/20
2 Novita 81.4 1443ms 1017ms 44.5 20/20
3 LLM Gateway 80.1 1986ms 2230ms 34.7 20/20
4 Concentrate AI 79.6 2571ms 2399ms 34.1 20/20
5 Vercel AI Gateway 78.8 2651ms 2437ms 34.8 20/20
6 Cloudflare AI Gateway 78.1 2821ms 2102ms 34.3 20/20
7 Kimi (direct) 77.6 2248ms 1690ms 35.2 19/20
8 OpenRouter 77.0 1907ms 1665ms 32.6 19/20
9 ngrok AI Gateway 75.8 2534ms 2399ms 34.9 19/20
10 blazerail 8.5 0ms 4447ms 127.1 2/20

View full run · SVG available as a build artifact

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant